Method, system and computer readable storage media for the detection of errors in three-dimensional measurements

ABSTRACT

A method, system and computer readable storage media for defects in images during three-dimensional measurement of teeth. An operator may use a dental camera to scan teeth and a trained deep neural network may automatically detect portions of the input images having defects and generate output information regarding those defects for corrective measures to be taken.

FIELD OF THE INVENTION

The present application relates generally to a method, a system andcomputer readable storage media for detecting errors inthree-dimensional (3D) measurements and, more particularly, to a method,system and computer readable storage media for utilizing deep learningmethods to detect errors in 3D measurements.

BACKGROUND OF THE INVENTION

Dental practitioners may be trained to detect errors in 3D images suchas intraoral images using the naked eye. These errors may bedefects/interferences in images caused by interference factors such as,for example, irregularities in intraoral scanner/camera's glass (e.g.scratches, fogging from breath, drops of saliva etc.), the presence offoreign objects (e.g. saliva bubbles or blood) on teeth being scanned,detrimental conditions in the measurement environment (e.g. sunlight orsurgical light) as well as malfunctioning of the intraoral scanner (e.g.pixel faults in the camera's sensor). If the errors are not recognizedduring the measurement/scanning process, the measurement may not becompleted, may have a low accuracy or may be noticeably delayed due torecurring interruptions.

However these errors as well as sources of the errors may be difficultto detect for both experienced and inexperienced dentists. Frequently,they may be noticed only at a later point in the measurement process orthrough conventional or manual checking of recording conditions. Morespecifically, some of the errors may be so subtle that conventionalmethods may not distinguish the errors from the general sensor noise andcalibration errors, which may always be present (E.g. small scratches oncamera glasses may cause small “bumps” in the measured data said “bumps”having heights similar to that of sensor noise. However, since the erroris systematic, it may not always be averaged out by additional images.Other, more severe errors, may indeed cause conventional checking torecognize a problem and then stop the acquisition process. However, aconventional/manual method may not be able to diagnose the cause of theerror and implementing filters with the required complexity using aconventional or manual approach is unmanageable.

An automatic detection of errors on the other hand may give an earlyindication of wrong measurements in order for suitable countermeasuresto be taken.

U.S. Pat. No. 9,788,917B2 discloses a method for employing artificialintelligence in automated orthodontic diagnosis and treatment planning.The method may include providing an intraoral imager configured to beoperated by a patient; receiving patient data regarding the orthodonticcondition; accessing a database that comprises or has access toinformation derived from orthodontic treatments; generating anelectronic model of the orthodontic condition; and instructing at leastone computer program to analyze the patient data and identify at leastone diagnosis and treatment regimen of the orthodontic condition basedon the information derived from orthodontic treatments.

U.S. Patent Application Publication No. 20190026893A1 discloses a methodfor assessing the shape of an orthodontic aligner wherein an analysisimage is submitted to a deep learning device, in order to determine avalue of a tooth attribute relating to a tooth represented on theanalysis image, and/or at least one value of an image attribute relatingto the analysis image.

PCT Application PCT/EP2018/055145 discloses a method for constructing arestoration in which a dental situation is measured by means of a dentalcamera and a three-dimensional (3D) model of the dental situation isgenerated. A computer-assisted detection algorithm may then be appliedto the 3D model of the dental situation and a type of restoration, atooth number or a position of the restoration are automaticallydetermined.

U.S. Application Publication No. 20180028294A1 discloses a method forDental CAD Automation using deep learning. The method may includereceiving a patient's scan data representing at least one portion of thepatient's dentition data set; and identifying, using a trained deepneural network, one or more dental features in the patient's scan.Herein, design automation may be carried out after complete scans havebeen generated. However this method does not improve the actual scanningprocess.

SUMMARY OF THE INVENTION

Existing limitations associated with the foregoing, as well as otherlimitations, can be overcome by a method, system and computer readablestorage media for utilizing deep learning methods to detect errors in 3Dmeasurements. These 3D measurements may include intraoral measurementsbut may not be limited to such measurements. For example, extraoralmeasurements such as measurements on plaster models may be included.

In an aspect herein, the present invention may provide a computerimplemented method for detecting defects during three-dimensionalmeasurements, the method comprising: receiving, by one or more computingdevices, individual images of a patient's dentition; automaticallyidentifying said defects using one or more output label values of atrained deep neural network by segmenting the individual images of thepatient's dentition into regions corresponding to semantic regionsand/or error regions, determining one or more corrective regimens tocorrect the identified defects, and combining the individual images ofthe patient's dentition to form a corrected global 3D image. The outputlabel values may be probability values.

In another aspect herein, the computer implemented method may furthercomprise one or more combinations of the following steps: (i) whereinthe individual images are individual three-dimensional optical images,(ii) wherein the individual images are received as a temporal sequenceof images, (iii) wherein the individual images comprise 3D measured dataand color data of the patient's dentition, (iv) wherein an indication ofa relevance of the identified error regions is based on correspondingsemantic regions, (v) further comprising: training the deep neuralnetwork using the one or more computing devices and a plurality ofindividual training images, to map one or more defects in at least oneportion of each training image to one or more probability values of aprobability vector, wherein the training is done on a pixel level byclassifying the individual training images and/or pixels of theindividual training images into one or more classes corresponding tosemantic data types and/or error data types, (vi) wherein the semanticdata types are selected from the group consisting of teeth, cheek, lip,tongue, gingiva, filling and ceramic and wherein the error data typesare selected from the group consisting of fogging, scratches, salivadroplets, dirt, blood, highlights, ambient lighting, measurementdistance, pixel faults, (vii) further comprising correcting the defectsby masking out locations corresponding to the defects prior toregistration of the individual images, (viii) further comprisingcorrecting the defects by partially including contributions of thelocations corresponding to the defects using predetermined weights, (ix)further comprising correcting the defects by automatically adjustingparameters of a dental camera corresponding to the defects, (x) whereinsaid parameters include exposure time, light intensity and temperatureof the dental camera's glass, (xi) further comprising indicating thedefects by relaying a warning to a user and/or generating a reportconcerning the error, (xii) wherein the deep neural network is a networkchosen from the group consisting of Convolutional Neural Networks (CNN),Recurrent Neural Networks (RNN) and Recurrent Convolutional NeuralNetworks (Recurrent-CNN).

In yet another aspect of the present invention, a non-transitorycomputer-readable storage medium storing a program may be provided,which, when executed by a computer system, causes the computer system toperform a procedure comprising: receiving, by one or more computingdevices, individual images of a patient's dentition; automaticallyidentifying defects in said individual images of the patient's dentitionusing one or more output probability values of a trained deep neuralnetwork by segmenting the individual images of the patient's dentitioninto regions corresponding to semantic regions and/or error regions,determining one or more corrective regimens to correct the identifieddefects and combining the individual images of the patient's dentitionto form a corrected global 3D image.

Further, an apparatus for detecting defects during three-dimensionalmeasurement, may be provided, the apparatus comprising a processorconfigured to: receive, by one or more computing devices, individualimages of a patient's dentition; automatically identify defects in saidindividual images of the patient's dentition using one or more outputlabel values of a trained deep neural network by segmenting theindividual images of the patient's dentition into regions correspondingto semantic regions and/or error regions, determine on or morecorrective regimens to correct the identified defects and combine theindividual images of the patient's dentition to form a corrected global3D image.

BRIEF DESCRIPTION OF THE DRAWINGS

Example embodiments will become more fully understood from the detaileddescription given herein below and the accompanying drawings, whereinlike elements are represented by like reference characters, which aregiven by way of illustration only and thus are not limitative of theexample embodiments herein and wherein:

FIG. 1 is a sketch of a top view of an oral cavity illustrating thescanning/recording of individual images of a patient's dentition.

FIG. 2 is a high level block diagram of a system according to anembodiment of the present invention.

FIG. 3 is a perspective view of a global 3D image of a dentition formedfrom individual images with their defects not removed according to anembodiment of the present invention.

FIG. 4 is a perspective view of a corrected global 3D image of adentition formed from individual images having their defects removedaccording to an embodiment of the present invention.

FIG. 5 is a high-level block diagram showing a structure of a neuralnetwork such as a deep neural network according to one embodiment.

FIG. 6A is a flow chart illustrating a method according to an embodimentof the present invention.

FIG. 6B is a sample image for training according to an embodiment of thepresent invention.

FIG. 7 is a block diagram illustrating an embodiment of the presentinvention.

FIG. 8 is another block diagram illustrating another embodiment of thepresent invention.

FIG. 9 is yet another block diagram illustrating an embodiment of thepresent invention.

FIG. 10 is a block diagram showing a computer system according to anexemplary embodiment of the present invention.

Different ones of the figures may have at least some reference numeralsthat may be the same in order to identify the same components, althougha detailed description of each such component may not be provided belowwith respect to each Figure.

DETAILED DESCRIPTION OF THE INVENTION

In accordance with example aspects described herein, a method, systemand computer readable storage media may be provided for utilizing deeplearning methods to detect errors in 3D measurements.

System for Detecting Errors in 3D Measurements

The accurate 3D measurement of a patient's oral cavity may be hinderedby factors such as saliva droplets, or blood on the patient's teeth. Thesystem described herein may preferably obtain images, such as individualthree-dimensional optical images 2 (FIG. 1), with each three-dimensionaloptical image 2 preferably comprising 3D measured data and color data ofa measured surface of the teeth and preferably being recordedsequentially in an oral cavity through a direct intraoral scan. This mayoccur, for example, in a dental office or clinic and may be performed bya dentist or dental technician. The images may also be obtainedindirectly through scanning an impression of the patient's teeth, orthrough a sequence of stored images. Using the images, preferablyobtained in a temporal sequence, a computer-implemented system mayautomatically identify and/or correct errors/defects 15 in the images inorder to facilitate accurate scanning of patient dentition. Herein, thecorrection may be done in real-time. Of course the images may also beindividual two-dimensional (2D) images, RGB Images, Range-Images(two-and-a-half-dimensional, 2.5D), 4-Channel Images (RGB-D), wheredepth and color may not be in perfect alignment, i.e. depth and colorimages may be acquired at different time. periods.

In the scanning process, a plurality of individual images may be createdand then a sequence 8 of at least two individual images or a pluralityof sequences 8 may be combined to form an overall/global 3D image 10(FIG. 3). More specifically, as shown in FIG. 1, individualthree-dimensional optical images 2, which are illustrated in the form ofrectangles, may be obtained by means of a scanner/dental camera 3 whichmay be moved relative to the object 1 along a measurement path 4 duringthe measurement. The dental camera 3 may be a handheld camera, forexample, which measures the object 1 using a fringe projection method.Other methods of 3D measurement may be realized by persons of ordinaryskill in the art. An overlapping area 5 between a first image 6 and asecond image 7, which is shown with a dashed line, is checked todetermine if recording conditions are met by using a computer and ifmet, the three-dimensional optical images 2 may be combined to form aglobal 3D image 10. The recording conditions may include an adequatesize, an adequate waviness, an adequate roughness, and/or an adequatenumber and arrangement of characteristic geometries. However, it may bedifficult to program the computer to determine errors/defects 15 in theindividual three-dimensional optical images 2 caused by interferencefactors in the same conventional way as the checking of the recordingconditions. A neural network on the other hand may learn complex tasksto recognize the cause of errors 15 and either isolate affected regions(so that the scanning process can proceed without interruption) or givea diagnosis along with suggested measures for correction to the user.Said interference factors may include irregularities in intraoralscanner/camera's glass (e.g. scratches, fogging from breath, drops ofsaliva etc.), the presence of foreign objects (e.g. saliva bubbles,blood, dirt) on teeth being scanned, detrimental conditions in themeasurement environment (e.g. sunlight or surgical light, measurementdistance/angle from teeth) as well as malfunctioning of the intraoralscanner (e.g. pixel faults in the camera's sensor, faulty LED). Saidinterference factors may also be considered as errors/defects 15themselves and may appear in the images. For example, defects 15 in theindividual three-dimensional optical images 2 caused by blood (which maybe similar to those caused by other fluids such as saliva droplets) mayappear as highlights and reflections in the individual three-dimensionaloptical images 2, leading to errors in the measurement. The defects 15caused by blood, for example, may be local to the regions covered by theblood. Thus, in removing the blood, the defects 15 may also be removed.Thus the terms defects 15 and errors may hereinafter be used tocollectively refer to defects/errors/interferences/distortions andinterference factors appearing in said individual three-dimensionaloptical images 2, which may at times be subtle enough to be missed bythe naked eye and may result in the inaccurate measurement of patientdentition, time consuming measurement of patient dentition, due to thepresence of a plurality of interruptions in the scanning process (thusrequiring repositioning of the scanning device), and/or impossiblemeasurement of the patient dentition (such as in the case of fogging ofthe camera glass).

The system may therefore train neural networks such as deep neuralnetworks, using a plurality of training data sets, to automaticallyrecognize and/or correct errors/defects 15 in the three-dimensionaloptical images 2, preferably in real time. Therefore, global defects 17propagated to the global 3D image 10 may be reduced or eliminated asshown in the corrected global 3D image 9 of FIG. 4 and/or the scan flowmay be improved due to fewer/no interruptions. For example, the presentsystem may detect and label defects 15 on a pixel level with labels thatcorrespond to fogging, scratches, saliva droplets, dirt, blood,highlights etc. or on an image level with labels such as ambientlighting, measurement distance etc. The present system may also identifyand label semantic data (for context purposes) with labels correspondingto teeth, cheek, lip, tongue, gingiva, filling, ceramic in thethree-dimensional optical images 2. Moreover, the system may determinecorrective measures and/or apply said determined corrective measuresupon detecting the errors, the errors being preferably determined in acontext aware manner (i.e. the context may be important to select anappropriate corrective method. E.g. saliva on cheek or gingiva may beignored, since the required accuracy of parts of the scan correspondingto the cheek may be much lower than that of the teeth).

FIG. 2 shows a block diagram of a system 200 for recognizing dentalinformation from individual three-dimensional optical images 2 ofpatients' dentitions according to one embodiment. System 200 may includea dental camera 3, a training module 204, an image correction module206, a computer system 100 and a database 202. In another embodiment,the database 202, image correction module 206, and/or training module204 may be part of the computer system 100 and/or may be able todirectly and/or indirectly adjust parameters of the dental camera 3based on a correction regimen. The computer system 100 may also includeat least one computer processor 122, a user interface 126 and input unit130. The computer processor may receive various requests and may loadappropriate instructions, as stored on a storage device, into memory andthen execute the loaded instructions. The computer system 100 may alsoinclude a communications interface 146 that enables software and data tobe transferred between the computer system 100 and external devices.

The computer system 100 may receive error detection requests from anexternal device such as the dental camera 3 or a user (not shown) andmay load appropriate instructions to detect said errors. Alternatively,the computer system may independently detect said errors upon receivingindividual three-dimensional optical images 2, without waiting for arequest.

In one embodiment, the computer system 100 may use many training datasets from a database 202 (which may include, for example, a plurality ofindividual three-dimensional optical images 2) to train one or more deepneural networks, which may be a part of training module 204. In someembodiments, system 200 may include a neural network module (not shown)that contains various deep learning neural networks such asConvolutional Neural Networks (CNN), Recurrent Neural Networks (RNN) andRecurrent Convolutional Neural Networks (Recurrent-CNN). An exampleRecurrent-CNN model is described in the publication by Courtney J.Spoerer et al, entitled “Recurrent Convolutional Neural Networks: ABetter Model of Biological Object Recognition” Front. Psychol., 12 Sep.2017, which is hereby incorporated by reference in its entirety, as ifset forth fully herein.

The training data sets and/or inputs to the neural networks may bepre-processed. For example, in order to process color data inconjunction with 3D measurements a calibration may be applied to aligncolor images with the 3D surface. Furthermore, standard dataaugmentation procedures such as synthetic rotations, scalings etc. maybe applied to the training data sets and/or inputs.

The training module 204 may use training data sets with labels tosupervise the learning process of the deep neural network. The labelsmay be used to describe a feature. The label values may be, for example,probability values of a probability vector. The training module 204 mayconversely use unlabeled training data sets to train generative deepneural networks.

The training data sets may be designed to train one or more deep neuralnetworks of training module 204 to identify the errors/defects 15. Forexample, to train a deep neural network to detect errors in theindividual three-dimensional optical images 2 caused by saliva droplets,a plurality of real life individual three-dimensional optical image datasets, having defects 15 caused by saliva droplets may be used as thetraining data sets specifically for saliva droplets. In another example,to train the deep neural network to recognize semantic data (e.g.,gingiva 11), another plurality of training data sets from real dentalpatients with one or more gingiva are selected to form a group oftraining data sets specifically for a gingiva. Database 202 maytherefore contain different groups of training data sets, one group foreach error data type and/or for each semantic data type, for example.

In an embodiment of the present invention, the training module 204 maytrain one or more deep neural networks in real-time. In someembodiments, training module 204 may pre-train one or more deep neuralnetworks using training data sets from database 202 such that thecomputer system 100 may readily use one or more pre-trained deep neuralnetworks to detect errors. It may then send information about thedetected errors and or the individual three-dimensional optical images2, preferably automatically and in real time, to an image correctionmodule 206 for correction of the detected errors. The detected errorsmay be corrected using predetermined correction regimens describedhereinafter and/or correction regimens obtained using artificialintelligence. For example, based on diagnostic data sets of pastcorrections with each diagnostic data set comprising detected errors,corresponding semantic data, corresponding corrections, as well as datafrom scientific literature, textbooks, input from users etc., the imagecorrection module 206 may be adapted to identify, recommend and/orimplement one or more correction regimens for the one or more detectederrors. The correction regimen may include, for example, computingcontributions to the final individual three-dimensional optical imagescoming from the sections having the errors, by using predeterminedweights. Systems for employing artificial intelligence in dentalplanning are described in U.S. Pat. No. 9,788,917B2, entitled “Methodsand systems for employing artificial intelligence in automatedorthodontic diagnosis and treatment planning” which is herebyincorporated by reference in its entirety, as if set forth fully herein.Of course, other non-artificial intelligence correction regimens such asnotifying a user, altering dental camera 3 parameters etc. may beemployed.

The database 202 may also store data related to the deep neural networksand the identified errors along with corresponding individualthree-dimensional optical images 2. Moreover, the computer system 100may have a display unit 126 and input unit 130 with which a user mayperform functions such as submitting a request and receiving andreviewing identified defects 15, etc. Other embodiments of the system200 may include different and/or additional components. Moreover, thefunctions may be distributed among the components in a different mannerthan described herein.

FIG. 5 shows a block diagram illustrating a structure of a neuralnetwork such as a deep neural network 300 according to an embodiment ofthe present invention. It may have several layers including an inputlayer 302, one or more hidden layers 304 and an output layer 306. Eachlayer may consist of one or more nodes 308, indicated by small circles.Information may flow from the input layer 302 to the output layer 306,i.e. left to right direction, though in other embodiments, it may befrom right to left, or both. For example, a recurrent network may takepreviously observed data into consideration when processing new data ina sequence 8 (e.g. current images may be segmented taking intoconsideration previous images), whereas a non-recurrent network mayprocess new data in isolation. A node 308 may have an input and anoutput and the nodes of the input layer 308 may be passive, meaning theymay not modify the data. For example, the nodes 308 of the input layer302 may each receive a single value (e.g. a pixel value) on their inputand duplicate the value to their multiple outputs. Conversely, the nodesof the hidden layers 304 and output layer 306 may be active, thereforebeing able to modify the data. In an example structure, each value fromthe input layer 302 may be duplicated and sent to all of the hiddennodes. The values entering the hidden nodes may be multiplied byweights, which may be a set of predetermined numbers associated witheach of the hidden nodes. The weighted inputs may then be summed toproduce a single number.

In an embodiment according to the present invention, the deep neuralnetwork 300 may use pixels of the individual three-dimensional opticalimages 2 as input when detecting some defects 15 such as fogging,scratches, saliva droplets, dirt, blood, highlights etc. The individualthree-dimensional optical images 2 may be color images. Herein, thenumber of nodes in the input layer 302 may be equal to the number ofpixels in an individual three-dimensional optical image 2. In an exampleembodiment, one neural network may be used for all defects 15 and inanother embodiment, different networks may be used for different defects15. In another example, the deep neural network 300 may classify theindividual three-dimensional optical images 2 instead of individualpixels when detecting some defects 15 such as those caused by ambientlight and measurement distance. In a further embodiment, the inputs maybe subsampled inputs, such as every 4^(th) pixel. In yet anotherembodiment, the deep neural network may have as inputs a plurality ofdata acquired by the dental camera 3 such as color-images, depthmeasurements, accelerations as well as device parameters such asexposure times, aperture etc. It may also incorporate a temporalsequence of the acquired data such as through employing a RecurrentConvolutional Neural Network (since some defects 15 may be difficult todetect using a single image). Defects 15 may in some cases be visiblemostly from a characteristic distortion that stays at the same imagelocations while the teeth are changing from image to image. This appliesto interference factors such as fogging, for example, as well as toscratches to a lesser extent. A recurrent network may be well suited torecognize such features in an image sequence 8. The deep neural networkmay output labels which may be, for example, a probability vector thatincludes one or more probability values of each pixel input belonging tocertain categories. For example, the output may contain a probabilityvector containing probability values wherein the highest probabilityvalues may define the defects 15. The deep neural network may alsooutput a map of label values without any attached probabilities.Moreover different classifications may be achieved. For example, a firstclassification may include one or more of defect categories, e.g.,scratches, fogging from breath, saliva bubbles, blood, dirt, sunlight,surgical light, measurement distance from teeth, pixel faults in thecamera's sensor etc. Another classification may include one or more ofsemantic categories e.g. teeth, cheek, lip, tongue, gingiva, filling,ceramic etc. A deep neural network can be created for eachclassification.

As discussed, the deep neural network may be a Convolutional NeuralNetwork (CNN), a Recurrent Neural Network (RNN), a RecurrentConvolutional Neural Network (Recurrent-CNN) or the like.

Method for Detecting Errors in 3D Measurements

Having described the system 200 of FIG. 2 reference will now be made toFIG. 6A, which shows a process S400 in accordance with at least some ofthe example embodiments herein.

The process S400 may begin by obtaining and marking areas of interest inthe training data sets with predetermined labels, Step S402. Forexample, sample defects 414 on sample image 412 (said sample image 414taken in a dark room and which would be black were there no interferencefactors) shown in FIG. 6B may be labelled as scratches. The marking ofthe training images may be done digitally e.g. by setting dots on theimages corresponding to the points of interest.

The training data may be labeled in order to assign semantics to theindividual three-dimensional optical images 2. This may happen on aper-pixel level for color or depth information. Alternatively, meshes ofcomplete 3D-models may be cut to compute corresponding per-pixel labelsfor single images. Moreover said meshes may be segmented such that thelabeling process may be automated. The meshes may be labelled and thelabels may be transferred to corresponding pixels of the single imagesin order to reduce the amount of work needed to label. These labels maydistinguish between teeth, cheek, lip, tongue, gingiva, filling, ceramicwhile assigning no label to anything else. Irrelevant for themeasurement may be cheek, lip, tongue and unlabeled data.

The training data may also be labeled in order to assign defect labelsto the individual three-dimensional optical images 2. This may also bedone on a per-pixel level for image or depth information. For example,the training data may be labeled on a pixel level for fogging,scratches, saliva droplets, dirt, blood, highlights and on an imagelevel for other information such as ambient lighting, measurementdistance, aperture etc.

The semantic labels may overlap with markers for defects 15, e.g. labelssuch as “Tooth+Saliva”, “Tooth+Blood”, “Tooth+Scratch” and “Tooth” maybe achieved, and these labels may be distinguishable from other labelssuch as “Cheek+Scratch”, i.e. saliva droplets on teeth (which may be arelevant defect) may be distinguished from saliva droplets on the cheek(which may be an irrelevant defect). This way, false notifications maybe avoided.

In an embodiment, certain efficiently computable filters such as imageprocessing filters including cross correlation, optical flow, edgedetectors, difference images and moving averages, may be applied to theinput data and the resulting filtered images with the same or lowerpixel resolution fed into the deep neural network 300 as additionalinput, in order to optimize the computational effort and increasenetwork practicability. The neural network's input layer may containadditional nodes in order process the additional per-pixel informationfrom the image filters.

Using this set of labeled or classified images, a deep neural network300 may be built and fed with the labeled images allowing the network to“learn” from it such that the network may produce a network wiring thatmay segment new images on its own.

As another option to segmentation on a per-image basis or on a per-pixelbasis, segmentation may involve classification on a level slightlyhigher than a per-pixel level (on a per “super-pixel” level, i.e.“super-pixels” are parts of images that are larger than normal pixels ofthe image).

Instructions and algorithms of process S400 may be stored in a memory ofthe computer system 100 and may be loaded and executed by processor 122to train (Step S404) one or more deep neural networks using the trainingdata sets to detect one or more defects 15 based on one or more outputprobability values of a probability vector. For example, if one of theprobability values of the probability vector that corresponds to ambientlight is 90%, then the neural network may detect excessive ambient lightas one of the defects 15 in the individual three-dimensional image. Inanother example, if a probability value corresponding to a location ofscratches is high, then the neural network identifies that correspondinglocation as the location of the scratches. Therefore, a deep neuralnetwork may be built and fed with the labeled images allowing thenetwork to “learn” from it such that the network may produce a networkwiring that may segment new images on its own.

The training may be done once, a plurality of times or intermittently.The training may also be semi- or self-supervised. For example, after afirst training, the deep neural network may receive or obtain previouslyunseen images and the output may be obtained, and corresponding feedbackmay be given such that the network may preferably operate on its owneventually to classify images without human help. This may be done on apixel level or on an image level. For example, on an image level, thedeep neural network 300 may be trained to map the images having ambientlight and images having no ambient light to probability vectors thathave probability values indicating the probabilities that the imageshave ambient light. The probability value indicating that the imageshave ambient light may therefore be the highest in the vectors.Therefore, the deep neural network 300 may be trained such that when asequence 8 of individual three-dimensional optical images 2 are inputinto the deep neural network 300, the deep neural network may returnresulting probability vectors for each image indicating the category(having ambient light or not) in which the images belongs.

After the training, the deep neural network may obtain or receive asequence 8 of individual three-dimensional optical images from a dentalcamera 3 to segment in real time (Step S406) and may detect the defects15 in the images (Step 408). Upon detecting said defects 15, thecorrection module 206 may correct (Step S410) the defects 15 and/orpropose corrective measures. This may be done in real time in sequentialimage registration (of the individual three-dimensional optical images)or in subsequent processing steps such as in global image registration(registration of all acquired images simultaneously 8), model surfacereconstruction and/or model texture generation. Correction may be doneby masking out unreliable data points corresponding to the defectlocations and/or using the data with lower weights so as to allowunaffected data points to override the errors introduced by theinterference factors.

FIG. 7 shows an embodiment of the present invention wherein the dentalcamera 3 records teeth 12 intraorally and generates a sequence 8 ofimages which include defects 15. The sequence 18 of images is input intothe deep neural network 300 which then detects the defects 15 and labelsthe locations 16 corresponding to the defects 15. In a next step, thelocations 16 are preferably ignored by masking them out prior toregistration of the images using a registration module 500. In anotherstep, a fully reconstructed triangle mesh/fully reconstructed 3Dimage/corrected global 3D image 9 may be obtained from the masked imagesusing a reconstruction module 502. Said reconstruction module 502 mayalso be used to generate the fully reconstructed triangle mesh 9 bypartially including contributions from the locations 16 corresponding tothe defects 15 through weighting means using predetermined weights. Saidreconstruction module may also be used to generate a colored texture forthe reconstructed triangle mesh 9 using the same weighting mechanism.

FIG. 8 shows another embodiment of the present invention, wherein thedeep neural network outputs information (such as a binary image showinglocations 16 of defects 15, or an indication of corrective measures tobe taken) or otherwise information such that said information is used bya controller 600 to automatically adjust parameters of the dental cameracorresponding to the detected defects 15 in order to reduce or eliminatefuture/incoming defects 15, e.g. based on the output, exposure time orlight intensity of the dental camera 3 may be automatically increased ordecreased and/or the camera's glass (or mirror, in some cases) may beheated to reduce/remove fogging. In an embodiment herein, the extent ofthe adjustment may be based on the relevance and/or size/amount of thedefect 15.

FIG. 9 shows yet another embodiment of the present invention, whereinthe deep neural network 300 issues information concerning a warning tothe user. The warning may be propagated to an acquisition unit 700 whichmay generate a report that may be accessed by a client such as customersupport via a network 702 such as a cloud network or internet, in orderto take corrective actions.

Computer System for Detecting Errors in 3D Measurements

Having described the process S400 of FIG. 6A reference will now be madeto FIG. 10, which shows a block diagram of a computer system 100 thatmay be employed in accordance with at least some of the exampleembodiments herein. Although various embodiments may be described hereinin terms of this exemplary computer system 100, after reading thisdescription, it may become apparent to a person skilled in the relevantart(s) how to implement the invention using other computer systemsand/or architectures.

The computer system 100 may include or be separate from the trainingmodule 204, database 202 and/or image correction module 206. The modulesmay be implemented in hardware, firmware, and/or software. The computersystem may also include at least one computer processor 122, userinterface 126 and input unit 130. The input unit 130 in one exemplaryembodiment may be used by the dentist along with a display unit 128 suchas a monitor to send instructions or requests about detecting defects15. In another exemplary embodiment herein, the input unit 130 is afinger or stylus to be used on a touchscreen interface (not shown). Theinput unit 130 may alternatively be a gesture/voice recognition device,a trackball, a mouse or other input device such as a keyboard or stylus.In one example, the display unit 128, the input unit 130, and thecomputer processor 122 may collectively form the user interface 126.

The computer processor 122 may include, for example, a centralprocessing unit, a multiple processing unit, an application-specificintegrated circuit (“ASIC”), a field programmable gate array (“FPGA”),or the like. The processor 122 may be connected to a communicationinfrastructure 124 (e.g., a communications bus, or a network). In anembodiment herein, the processor 122 may receive a request for 3Dmeasurement and may automatically detect defects 15 in the images,automatically correct said defects 15 in the images using the trainingmodule 204, database 202 and image correction module 206. The processor122 may achieve this by loading corresponding instructions stored in anon-transitory storage device in the form of computer-readable programinstructions and executing the loaded instructions.

The computer system 100 may further comprise a main memory 132, whichmay be a random access memory (“RAM”) and also may include a secondarymemory 134. The secondary memory 134 may include, for example, a harddisk drive 136 and/or a removable-storage drive 138. Theremovable-storage drive 138 may read from and/or write to a removablestorage unit 140 in a well-known manner. The removable storage unit 140may be, for example, a floppy disk, a magnetic tape, an optical disk, aflash memory device, and the like, which may be written to and read fromby the removable-storage drive 138. The removable storage unit 140 mayinclude a non-transitory computer-readable storage medium storingcomputer-executable software instructions and/or data.

In further alternative embodiments, the secondary memory 134 may includeother computer-readable media storing computer-executable programs orother instructions to be loaded into the computer system 100. Suchdevices may include a removable storage unit 144 and an interface 142(e.g., a program cartridge and a cartridge interface); a removablememory chip (e.g., an erasable programmable read-only memory (“EPROM”)or a programmable read-only memory (“PROM”)) and an associated memorysocket; and other removable storage units 144 and interfaces 142 thatallow software and data to be transferred from the removable storageunit 144 to other parts of the computer system 100.

The computer system 100 also may include a communications interface 146that enables software and data to be transferred between the computersystem 100 and external devices. Such an interface may include a modem,a network interface (e.g., an Ethernet card, a wireless interface, acloud delivering hosted services over the internet, etc.), acommunications port (e.g., a Universal Serial Bus (“USB”) port or aFireWire® port), a Personal Computer Memory Card InternationalAssociation (“PCMCIA”) interface, Bluetooth®, and the like. Software anddata transferred via the communications interface 146 may be in the formof signals, which may be electronic, electromagnetic, optical or anothertype of signal that may be capable of being transmitted and/or receivedby the communications interface 146. Signals may be provided to thecommunications interface 146 via a communications path 148 (e.g., achannel). The communications path 148 may carry signals and may beimplemented using wire or cable, fiber optics, a telephone line, acellular link, a radio-frequency (“RF”) link, or the like. Thecommunications interface 146 may be used to transfer software or data orother information between the computer system 100 and a remote server orcloud-based storage.

One or more computer programs or computer control logic may be stored inthe main memory 132 and/or the secondary memory 134. The computerprograms may also be received via the communications interface 146. Thecomputer programs may include computer-executable instructions which,when executed by the computer processor 122, cause the computer system100 to perform the methods as described herein.

In another embodiment, the software may be stored in a non-transitorycomputer-readable storage medium and loaded into the main memory 132and/or the secondary memory 134 of the computer system 100 using theremovable-storage drive 138, the hard disk drive 136, and/or thecommunications interface 146. Control logic (software), when executed bythe processor 122, causes the computer system 100, and more generallythe system for detecting scan interferences, to perform all or some ofthe methods described herein.

Implementation of other hardware and software arrangement so as toperform the functions described herein will be apparent to personsskilled in the relevant art(s) in view of this description.

What is claimed is:
 1. A method comprising: receiving, by one or morecomputing devices, individual images of a patient's dentition as atemporal sequence; providing the individual images as input to a traineddeep neural network; automatically identifying in the individual imagesdefects due to one or more interference factors using one or more outputlabel values of the trained deep neural network, said defects beingcharacteristic of inaccurate measurement of patient dentition, timeconsuming measurement of patient dentition relative to a measurementwithout said defects and/or impossible measurement of patient dentition,said automatically identifying being performed by the trained neuralnetwork segmenting the individual images of the patient's dentition intoregions corresponding to semantic regions and/or error regions,determining one or more corrective regimens to correct the automaticallyidentified defects in a context aware manner, an indication of arelevance of the error regions for the context aware manner being basedon corresponding semantic regions; correcting said automaticallyidentified defects in said context aware manner; and combining,responsive to said correcting, the individual images of the patient'sdentition to form a corrected global 3D image.
 2. The method accordingto claim 1, wherein the individual images are individualthree-dimensional optical images.
 3. The method according to claim 1,wherein the individual images comprise 3D measured data and color dataof the patient's dentition.
 4. The method according to claim 1, furthercomprising: training the deep neural network using the one or morecomputing devices and a plurality of individual training images, to mapone or more defects in at least one portion of each training image toone or more label values, wherein the training is done on a pixel levelby classifying the individual training images and/or pixels of theindividual training images into one or more classes corresponding tosemantic data types and/or error data types.
 5. The method according toclaim 4, wherein the semantic data types are selected from the groupconsisting of teeth, cheek, lip, tongue, gingiva, filling and ceramicand wherein the error data types are selected from the group consistingof fogging, scratches, saliva droplets, dirt, blood, highlights, ambientlighting, measurement distance, pixel faults.
 6. The method according toclaim 1, further comprising correcting the defects by masking outlocations corresponding to the defects prior to registration of theindividual images.
 7. The method according to claim 1, furthercomprising correcting the defects by partially including contributionsof the locations corresponding to the defects using predeterminedweights.
 8. The method according to claim 1, further comprisingcorrecting the defects by automatically adjusting parameters of a dentalcamera corresponding to the defects.
 9. The method according to claim 8,wherein said parameters include exposure time, light intensity andtemperature of the dental camera's glass.
 10. The method according toclaim 1, further comprising indicating the defects by relaying a warningto a user and/or generating a report concerning the error.
 11. Themethod according to claim 1, wherein the deep neural network is anetwork chosen from the group consisting of Convolutional NeuralNetworks (CNN), Recurrent Neural Networks (RNN) and RecurrentConvolutional Neural Networks (Recurrent-CNN).
 12. The method accordingto claim 1, wherein the individual images are individual two-dimensional(2D) images.
 13. A non-transitory computer-readable storage mediumstoring a program which, when executed by a computer system, causes thecomputer system to perform a procedure comprising: receiving, by one ormore computing devices, individual images of a patient's dentition as atemporal sequence; providing the individual images as input to a traineddeep neural network; automatically identifying in the individual imagesdefects due to one or more interference factors using one or more outputlabel values of the trained deep neural network, said defects beingcharacteristic of inaccurate measurement of patient dentition, timeconsuming measurement of patient dentition relative to a measurementwithout said defects and/or impossible measurement of patient dentition,said automatically identifying being performed by the trained neuralnetwork segmenting the individual images of the patient's dentition intoregions corresponding to semantic regions and/or error regions,determining one or more corrective regimens to correct the automaticallyidentified defects in a context aware manner, an indication of arelevance of the error regions for the context aware manner being basedon corresponding semantic regions; correcting said automaticallyidentified defects in said context aware manner; and combining,responsive to said correcting, the individual images of the patient'sdentition to form a corrected global 3D image.
 14. A system fordetecting defects during three-dimensional measurement, comprising aprocessor configured to perform the steps comprising: receiving, by oneor more computing devices individual images of a patient's dentition asa temporal sequence; providing the individual images as input to atrained deep neural network; automatically identifying in the individualimages defects due to one or more interference factors using one or moreoutput label values of the trained deep neural network, said defectsbeing characteristic of inaccurate measurement of patient dentition,time consuming measurement of patient dentition relative to ameasurement without said defects and/or impossible measurement ofpatient dentition, said automatically identifying being performed by thetrained neural network segmenting the individual images of the patient'sdentition into regions corresponding to semantic regions and/or errorregions, determining one or more corrective regimens to correct theautomatically identified defects in a context aware manner, anindication of a relevance of the error regions for the context awaremanner being based on corresponding semantic regions; correcting saidautomatically identified defects in said context aware manner; andcombining, responsive to said correcting, the individual images of thepatient's dentition to form a corrected global 3D image.
 15. The systemaccording to claim 14, wherein the deep neural network is a networkchosen from the group consisting of Convolutional Neural Networks (CNN),Recurrent Neural Networks (RNN) and Recurrent Convolutional NeuralNetworks (Recurrent-CNN).